<html>
<head><meta charset="utf-8"><title>Meta? About perf runs infra · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html">Meta? About perf runs infra</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="202277194"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202277194" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202277194">(Jun 29 2020 at 06:57)</a>:</h4>
<p>Where can I ask about the infrastructure where perf runs are run?</p>
<p>I'm interested in what is being worked on there. Specifically, I'd like to know if there are plans to have more than one particular hardware setup in the future, so that the runs have more representative results. </p>
<p>Among other things ^^</p>



<a name="202306348"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202306348" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202306348">(Jun 29 2020 at 13:10)</a>:</h4>
<p>no we probably won't have multiple hardware platforms just for the sake of "representative" runs -- that's not really what our perf infra focuses on today I think</p>



<a name="202306370"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202306370" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202306370">(Jun 29 2020 at 13:10)</a>:</h4>
<p>but you can ask here and I can try to answer</p>



<a name="202344219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202344219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202344219">(Jun 29 2020 at 17:39)</a>:</h4>
<p>What does it focus on? <span class="user-mention" data-user-id="116122">@simulacrum</span></p>



<a name="202344313"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202344313" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202344313">(Jun 29 2020 at 17:40)</a>:</h4>
<p>Uh well I don't really know that there's a good descriptor</p>



<a name="202345118"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202345118" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202345118">(Jun 29 2020 at 17:47)</a>:</h4>
<p>So here is my reasoning:</p>
<p>I ask because measuring software performance with any degree of certainty is hard, and moreso when you want your samples to be representative of other computers.</p>
<p>If we're thus making decisions based on the perf benchmarks that we run, we should strive to make them representative of the realities of the Rust community.</p>
<p>The community has a very diverse setup of machines with which they work. If our benchmarks are run on just one machine, we need to know if it is actually representative of the whole. </p>
<p>Does the amount of RAM change performance? Does the amount of cores change it? Does the size of the cpu cache change it?</p>
<p>If any of the answers to those questions is Yes, then we should really think about why we have only one machine, shouldn't we? :)</p>



<a name="202345233"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202345233" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202345233">(Jun 29 2020 at 17:48)</a>:</h4>
<p>That's basically where I came from to ask that question</p>



<a name="202345456"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202345456" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202345456">(Jun 29 2020 at 17:50)</a>:</h4>
<p>I thought that what was limiting us from having more machines was money, but I don't really know. Is it the cost? How much does running the perf benchmarks cost per month, more or less?<br>
Or is it something different? Is it the burden of maintaining it that's keeping us out of that?</p>



<a name="202345635"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202345635" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202345635">(Jun 29 2020 at 17:52)</a>:</h4>
<p>I'm hearing "I'd like to take over maintenance and development of perf.rlo" <span aria-label="smirk" class="emoji emoji-1f60f" role="img" title="smirk">:smirk:</span></p>



<a name="202345712"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202345712" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> bjorn3 <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202345712">(Jun 29 2020 at 17:53)</a>:</h4>
<blockquote>
<p>Does the amount of RAM change performance?</p>
</blockquote>
<p>I think only when having little ram would cause swapping.</p>
<blockquote>
<p>Does the amount of cores change it?</p>
</blockquote>
<p>check builds: no, those are single threaded within a single rustc process.<br>
debug/opt builds: yes, optimization after codegen happens in parallel</p>
<blockquote>
<p>Does the size of the cpu cache change it?</p>
</blockquote>
<p>Probably</p>



<a name="202346668"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202346668" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202346668">(Jun 29 2020 at 18:00)</a>:</h4>
<blockquote>
<p>I'm hearing "I'd like to take over maintenance and development of perf.rlo" <span aria-label="smirk" class="emoji emoji-1f60f" role="img" title="smirk">:smirk:</span></p>
</blockquote>
<p><span class="user-mention" data-user-id="116155">@Jake Goulding</span> I really kind of do, dammit XD you figured me out.<br>
I might or might not be able to in the short term. I have promised away 4 months of me to the mir-opt group through my undergrad thesis, which will hopefully get started in July.<br>
Considering where the world is and where I am right now, I don't think I have the bandwidth. But heck, after I clear the thesis? I would be SO HAPPY TO DO IT. There aren't many things that call to me as much as testing and measuring do. <span aria-label="blush" class="emoji emoji-1f60a" role="img" title="blush">:blush:</span></p>



<a name="202347039"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347039" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347039">(Jun 29 2020 at 18:03)</a>:</h4>
<p><span class="user-mention" data-user-id="133247">@bjorn3</span> thank you for the answers :)<br>
I'm guessing that cache size matters a lot up to a point, after which caching doesn't help much. It would be nice to test that. I wonder if there are still chips where model A and model B only differ in their cache size?</p>



<a name="202347508"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347508" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347508">(Jun 29 2020 at 18:07)</a>:</h4>
<blockquote>
<p>I thought that what was limiting us from having more machines was money, but I don't really know. Is it the cost? How much does running the perf benchmarks cost per month, more or less?<br>
Or is it something different? Is it the burden of maintaining it that's keeping us out of that?</p>
</blockquote>
<p>So, at the moment we're running benchmarks on Hetzner's AX41-NVMe: <a href="https://www.hetzner.com/dedicated-rootserver/ax41">https://www.hetzner.com/dedicated-rootserver/ax41</a></p>



<a name="202347658"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347658" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347658">(Jun 29 2020 at 18:08)</a>:</h4>
<p>the "problem" is that we can't use VMs for benchmarks, so we can't rely on our existing sponsorships (spinning up a new VM on AWS would be way easier) and we need to seek the money somewhere, and we're limited on where we can get the servers at (because we're limited on bare metals)</p>



<a name="202347839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347839">(Jun 29 2020 at 18:10)</a>:</h4>
<p>Right, right, that makes a lot of sense</p>



<a name="202347880"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347880" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347880">(Jun 29 2020 at 18:10)</a>:</h4>
<p>So it is a bit of both columns then</p>



<a name="202347910"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347910" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347910">(Jun 29 2020 at 18:11)</a>:</h4>
<p>Money + "wait we need to run this on bare metal, heck"</p>



<a name="202347921"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347921" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347921">(Jun 29 2020 at 18:11)</a>:</h4>
<p>yeah</p>



<a name="202347945"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347945" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347945">(Jun 29 2020 at 18:11)</a>:</h4>
<p>and well, some impl work will need to be done :P</p>



<a name="202347949"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347949" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347949">(Jun 29 2020 at 18:11)</a>:</h4>
<p>Yas</p>



<a name="202347955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347955">(Jun 29 2020 at 18:11)</a>:</h4>
<p>:P</p>



<a name="202347964"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202347964" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202347964">(Jun 29 2020 at 18:11)</a>:</h4>
<p>Of course</p>



<a name="202355781"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202355781" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202355781">(Jun 29 2020 at 19:15)</a>:</h4>
<p>there's also the element that personally I think we're not really at the point where representative sampling makes a lot of sense</p>



<a name="202355877"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202355877" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202355877">(Jun 29 2020 at 19:16)</a>:</h4>
<p>even just the one machine we have now gives us a ton of data per run</p>



<a name="202355955"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202355955" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202355955">(Jun 29 2020 at 19:17)</a>:</h4>
<p>and to my knowledge, focusing on instruction counts is going to be pretty deterministic across machines independent of cache size/memory size/disk size, or at least mostly so</p>



<a name="202355976"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202355976" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202355976">(Jun 29 2020 at 19:17)</a>:</h4>
<p>(obviously there's differences, but ... not significant, I'd imagine)</p>



<a name="202356120"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202356120" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202356120">(Jun 29 2020 at 19:18)</a>:</h4>
<p>I would also say that -- at least for now -- we don't seem to be at the point where we're "fast enough" on the <em>one</em> machine we do have</p>



<a name="202356151"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202356151" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202356151">(Jun 29 2020 at 19:18)</a>:</h4>
<p>so we're not really at the point where we're making decisions like "well this would be better here but worse on this other machine/architecture"</p>



<a name="202356219"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202356219" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202356219">(Jun 29 2020 at 19:19)</a>:</h4>
<p>of course, maybe that's partially for a lack of data... but I think at least for the most part rustc is going to be pretty equivalent independent of machine (possibly modulo things like windows antivirus, but that's not really about the hardware)</p>



<a name="202356356"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202356356" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202356356">(Jun 29 2020 at 19:20)</a>:</h4>
<p>and I think if I was to suggest where to spend time, it would be in increasing the usability of the metrics we do gather and expanding that set -- we mostly just look at instructions today, because everything else is quite variable. But there's a lot of other interesting metrics to collect</p>



<a name="202356374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202356374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202356374">(Jun 29 2020 at 19:20)</a>:</h4>
<p>one project that IMO would be worthwhile is figuring out a way for the self-profile data to come with instruction counts</p>



<a name="202357224"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202357224" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202357224">(Jun 29 2020 at 19:29)</a>:</h4>
<blockquote>
<p>deterministic across machines</p>
</blockquote>
<p>I do wonder about different architectures. E.g. with the impending ARMpocalypse, I could see codegen changes being good on x86 and bad on ARM (or vice versa). That should be a smaller side project to figure that out, however.</p>



<a name="202362682"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202362682" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202362682">(Jun 29 2020 at 20:18)</a>:</h4>
<p>You expose many fair points, <span class="user-mention" data-user-id="116122">@simulacrum</span>.</p>
<p>The other thing I would like to see is less latency, by having more than one machine available (this time, not in the sense of "different machines", but actually in the sense of "more copies of the same reference machine"). I think we could really use a second one at the moment, because there are some days where you have to wait around 24 hours for a perf run to be done.</p>



<a name="202362735"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202362735" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202362735">(Jun 29 2020 at 20:19)</a>:</h4>
<p>Yeah, different archs are interesting to look at (more so than anything else). Not really feasible for our CI budget though.</p>



<a name="202362787"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202362787" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202362787">(Jun 29 2020 at 20:20)</a>:</h4>
<p>hm, so it's true that less latency is perhaps somewhat desirable, but the intent is that people can run it locally, and that gives you <em>very</em> low latency</p>



<a name="202362889"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202362889" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202362889">(Jun 29 2020 at 20:20)</a>:</h4>
<p>we can't do much better than try builds + ~2 hours anyway, and that gets to around 4 hours. tbh, I usually can't iterate more than twice per day at that rate... so once or twice doesn't matter much</p>



<a name="202363543"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/Meta%3F%20About%20perf%20runs%20infra/near/202363543" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Félix Fischer <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra.html#202363543">(Jun 29 2020 at 20:26)</a>:</h4>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra/near/202356356">said</a>:</p>
<blockquote>
<p>and I think if I was to suggest where to spend time, it would be in increasing the usability of the metrics we do gather and expanding that set -- we mostly just look at instructions today, because everything else is quite variable. But there's a lot of other interesting metrics to collect</p>
</blockquote>
<p>I completely agree :)<br>
Both w.r.t. usability (How do you answer "where should we spend most of our resources if we want to increase compilation performance?" and "what kind of metrics are we currently lacking?")<br>
And also w.r.t. interesting metrics that we could be collecting :3</p>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra/near/202355976">said</a>:</p>
<blockquote>
<p>(obviously there's differences, but ... not significant, I'd imagine)</p>
</blockquote>
<p>I would like to test that, just to be sure we're standing on more or less solid assumptions.</p>
<p><span class="user-mention silent" data-user-id="116122">simulacrum</span> <a href="#narrow/stream/242791-t-infra/topic/Meta.3F.20About.20perf.20runs.20infra/near/202356374">said</a>:</p>
<blockquote>
<p>one project that IMO would be worthwhile is figuring out a way for the self-profile data to come with instruction counts</p>
</blockquote>
<p>Ohhh, yeah, that would be really nice. If this was solid enough, it could open the door for "benchmarking rustc yourself". If given enough control over what tests to run and so on (because not all tests will be relevant to all possible modifications to the compiler), it could really be a quick way of testing your own changes for performance without having to rely on the bandwidth of perf.rlo</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>